Training and Serving ML workloads with Kubeflow at CERN

Machine Learning (ML) has been growing in popularity multiple areas and groups at CERN, covering fast simulation, tracking, anomaly detection, among many others. We describe a new service available based on Kubeflow managing the full ML lifecycle: data preparation interactive analysis, large scale distributed model training serving. cover specific features for hyper-parameter tuning metadata management, as well infrastructure details to integrate accelerators external resources. also present results cost evaluation from scaling out popular use case using public cloud resources, achieving close linear when number of GPUs.